Add Init Hooks: Auto-Apply Seed Fixtures on Boot (#335 P1) - #339
Conversation
Bring the emulator up in a known state without manual seeding: drop *.json seed fixtures in an init directory and they're applied on every startup. - serve --init-dir <dir>: after providers are built and any persisted state is restored, before serving, apply every *.json (lexical order) as a seed fixture to all running providers. A parse error fails startup; a per-provider apply error (e.g. a resource that already exists from restored state) logs a warning and boot continues. - lifecycle start auto-loads <run-dir>/init.d when it exists (drop-in), unless the user passes an explicit --init-dir. Fixtures are provider-agnostic, so one file seeds S3/Blob/GCS alike. Running setup scripts on boot is a planned follow-up.
NitinKumar004
left a comment
NitinKumar004
left a comment
There was a problem hiding this comment.
Deep review — init hooks
Small, clean, well-placed feature. Verified in an isolated worktree at the PR head; gates green (build + GOOS=windows go build ./cmd/cloudemu / vet / go test -race ./cmd/cloudemu/ ok / gofmt clean). Applied at the right point (after build + persistence-restore, before serving), iterates all providers per the provider-agnostic fixture model, no concurrency concern (boot is single-threaded before Serve), and start auto-loads <run-dir>/init.d unless --init-dir is given.
Holding on one Medium (silent-correctness) plus a Low note.
M1 (Medium) — a colliding resource silently truncates the rest of a fixture
applyInitFile warns-and-continues per file, but seed.Apply is category-level fail-fast: applyBuckets returns on the first CreateBucket error, and Apply then skips all tables/secrets/instances. So the first already-exists resource aborts everything after it in that fixture, while boot proceeds reporting only a warning. Two realistic silent-loss paths:
- Two overlapping init files, no
--persist:00-base.jsoncreates bucketX;01-app.jsonhas bucketX(dup) + tableY→Yis never created, boot looks fine. --persist+init.doverlap: restored state collides with a fixture's early resource → any new resource later in that same file is silently dropped on restart.
The "duplicate → warns not fails" policy (and TestApplyInitDirDuplicateWarnsNotFails, which has a single bucket) implies graceful per-resource skipping that isn't there — a duplicate truncates the fixture from that point. Fix: apply init idempotently at resource granularity (skip AlreadyExists and keep going — e.g. a best-effort / IgnoreExisting mode on seed.Apply, or pre-filter existing resources). At minimum, document that a fixture aborts at the first existing resource, and add a test asserting a post-collision resource still applies. Inline below.
L1 (Low) — gosec G703 on os.Stat(path) (lifecycle.go:77)
Taint from the --home-derived path — same linter-version drift as the earlier lifecycle PRs (newer bundled gosec; your --new-from-rev reports 0), and it's the user's own --home, not a trust boundary. Non-blocking; a //nolint:gosec closes it if the pinned linter ever flags it.
No AI attribution. Requesting changes on M1; L1 is a non-blocking note.
| } | ||
|
|
||
| for prov, t := range targets { | ||
| if err := seed.Apply(ctx, f, t); err != nil { |
There was a problem hiding this comment.
M1 (Medium). seed.Apply is category-level fail-fast — applyBuckets returns on the first CreateBucket that errors (e.g. AlreadyExists), and Apply then never reaches tables/secrets/instances. Since this loop only warns per provider/file, the first colliding resource silently drops every resource after it in the fixture, yet boot continues "successfully." Triggers without any exotic setup: two init files that overlap on an early resource (bucket X in 00, then X+table Y in 01 → Y lost), or --persist restoring a resource that an init fixture re-declares before a new one. Make init application idempotent at resource granularity — skip AlreadyExists and continue (a best-effort/IgnoreExisting mode on seed.Apply, or pre-filter what already exists) — so a duplicate skips just that resource, matching the documented "warn and continue." Please also extend TestApplyInitDirDuplicateWarnsNotFails with a second resource after the duplicate and assert it still gets created.
There was a problem hiding this comment.
Fixed. seed.Apply now takes an IgnoreExisting() option that skips a resource returning AlreadyExists and continues, instead of the category-level fail-fast that truncated everything after the first collision. Init application (applyInitFile) uses it, so a duplicate skips just that resource and the rest of the fixture still applies — matching the documented "warn and continue". A bucket/table that already exists still has its declared objects/items applied; secrets/instances that exist are skipped.
Extended TestApplyInitDirDuplicateWarnsNotFails: the fixture is now bucket dup (pre-created → collision) followed by table fresh, and the test asserts fresh is created despite the collision. Also E2E-verified: --persist restores a bucket, init.d re-declares it (collision), and a table declared after it is still created on restart. 0c12468
…eview) A duplicate resource previously truncated the rest of a fixture: seed.Apply is category-level fail-fast, so the first AlreadyExists aborted everything after it while boot only logged a warning — silent data loss when two init files overlap or an init fixture re-declares restored state. - seed.Apply gains an IgnoreExisting option that skips AlreadyExists per resource and continues, instead of failing the whole fixture. - Init application uses it, so a colliding resource skips just itself and the rest of the fixture still applies. - Extend the duplicate test with a resource after the collision, asserting it is still created.
|
Thanks for the review — M1 fixed in
Gate green: build / vet / gofmt / |
NitinKumar004
left a comment
NitinKumar004
left a comment
There was a problem hiding this comment.
Approving — M1 resolved
Verified the fix (0c124680) in an isolated worktree at the PR head; gate green (build + GOOS=windows go build ./cmd/cloudemu / vet / go test -race ./seed/... ./cmd/cloudemu/ ok / gofmt clean).
M1 (a colliding resource truncating the rest of a fixture) is fully resolved, and cleanly:
seed.Applygains an additive variadic...OptionwithIgnoreExisting(). The two production callers are exactly right — the/seedendpoint (serve.go:253) passes no opts and keeps its old fail-fast behavior, while init (serve.go:512) opts in — so persist-restore and the seed admin path have zero blast radius from the shared-package change.- On an
AlreadyExists(detected viacerrors.IsAlreadyExists), each ofapplyBuckets/applyTables/applySecrets/applyInstancesskips just that resource and continues, and buckets/tables still (re)put their declared objects/items. Genuine PutItem/PutObject errors still fail hard. TestApplyInitDirDuplicateWarnsNotFailsnow uses a fixture where the first resource collides and a new one (fresh) follows, assertingfreshis still created — exactly the post-collision-still-applies case that was missing.
One non-blocking Low remains: gosec G703 on lifecycle.go:77 (os.Stat on the --home-derived path) is the same linter-version-drift false-positive as the earlier lifecycle PRs — a //nolint:gosec closes it if your pinned linter ever flags it.
Clean, well-architected fix with a proving test. LGTM.
2 participants
Objective / Issue
P1 item of the "minikube for cloud resources" roadmap (#335). Bring the emulator up in a known state without manual seeding: drop
*.jsonseed fixtures in an init directory and they're applied on every startup (docker-entrypoint.d-style).What we found
seed.Load/seed.Applyalready turn a declarative fixture into resources through the provider-agnostic driver interfaces, andservealready has a clean boot sequence (build providers → restore persistence → serve). So init hooks are a thin layer: read a directory of fixtures at the right point in boot.--init-dir(and noinit.d), behavior is unchanged.How we fixed it / How it works
serve --init-dir <dir>: after providers are built and any persisted state is restored, before serving, apply every*.json(lexical order) as aseed.Fixturesto all running providers. Fixtures are provider-agnostic, so one file seeds S3/Blob/GCS alike.cloudemu startauto-loads<run-dir>/init.dwhen it exists (drop-in), unless the user passes an explicit--init-dir.sequenceDiagram actor U as You participant S as serve (boot) participant D as init dir participant P as providers U->>D: drop 01-baseline.json (buckets/tables/secrets) U->>S: cloudemu start S->>S: build providers, restore persistence S->>D: read *.json (lexical order) S->>P: seed.Apply each fixture to every provider S-->>U: serving, already in the boot stateAlternatives not taken
*.shrun post-ready with endpoint env) — the powerful but exec/security-heavy half of init hooks; deferred to a follow-up to keep this a small, safe change. Noted in the docs.Docs / Tests / Playground
docs/standalone-server.md— init-hooks section (drop-in usage, ordering, failure policy, scripts-deferred note).Test plan
go test -race ./cmd/cloudemu/...go test ./.../go mod tidy/ golangci-lint = clean (CodeQL: only pre-existing findings, none in changed files).cloudemu start+ realawsCLI on the default~/.cloudemu: droppedinit.d/01-baseline.json(bucket+object, table+item, secret) →start→ emulator booted straight into that state (config.yaml=env: local, DynamoDB itemAda, secrets3cr3t) with no manual seeding; restart re-applies cleanly.Risk & Rollback
--init-dir/ noinit.d= unchanged). Malformed fixtures fail fast with a clear error; apply errors are non-fatal warnings.Conclusion
cloudemu startnow brings the local cloud up pre-seeded from a drop-ininit.d, reusing the existing fixture engine.Follow-up (#335): executable boot scripts (post-ready, endpoint env) for setup beyond the fixture schema.